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Abstract 

Twenty years ago, John Cannell developed data on test scores that became known as the "Lake 
Wobegon effect." This commentary describes that experience. 


Almost twenty years ago, I wrote - and then privately published - the two "Lake 
Woebegone" reports, named after Garrison Keillor's mythical Minnesota town 
where "all the women are strong, all the men are good-looking, and all the 
children are above average." The first "Lake Woebegone" report documented 
that all fifty states were testing above the national average in elementary 
achievement and concluded the testing infrastructure in America's public 
schools was corrupt. 1 The second report delineated the systematic and pervasive 
ways that American educators cheat on standardized achievement tests. 2 Both 
reports received widespread national publicity, were extensively discussed in 
academic journals, and helped spur the testing reform movement. 

In 1991, five years after I began, I abruptly left the testing reform movement. 

This paper discusses how I learned about "Lake Woebegone" testing, the reason 
why I left the testing reform movement, and my observations on where testing is 
today. Is No Child Left Behind (NCLB) testing much different from what was 
occurring during the "Lake Woebegone" years? 

I continue to assume that tests are simply sampling techniques, in which a 
relatively small number of test questions sample a larger body of knowledge. 
Any corruption of the sampling technique, in which the test questions become 
more familiar to students than the larger body of knowledge being sampled, 
invalidates the test. As always, my observations are those of a consumer, a 
parent, a physician, and an activist - not those of a psychometrician. 

My education about the corruption of American public school achievement 
testing was a gradual process. It started in my medical office in a tiny town in 
the coal fields of Southern West Virginia, led to school rooms in the county and 
then the state, to the offices of testing directors and school administrators around 
the country, to the boardrooms of commercial test publishers, to the office of the 
U.S. Secretary of Education, to schools of education at major American 
universities, to various governors' offices, and finally, to two American 
presidents. 

The author wishes to acknowledge Richard Phelps, who disagrees my work was for naught. 
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One day in 1985, West Virginia newspapers announced all fifty-five West 
Virginia counties had tested above the national average. Amid the mutual 
congratulations, I asked myself two things. How could all the counties in West 
Virginia, perhaps the poorest and most illiterate state in the union, be above the 
national average? Moreover, if West Virginia was above average, what state was 
below? 

In my Flat Top, West Virginia, clinic, illiterate adolescent patients with 
unbelievably high standardized achievement test scores told me their teachers 
drilled them on test questions in advance of the test. How did the teachers know 
what questions would be on a standardized test? 

Then I learned that West Virginia schools, like most other states, used what 
seemed to me as a physician to be very unusual standardized tests. Unlike the 
standardized tests that I knew - such as college entrance, medical school 
admission, or medical licensure examinations - public school achievement exams 
used the same exact questions year after year and then compared those scores to 
an old, and dubious, norm group - not to a national average. Furthermore, 
educators - the group really being tested - had physical control of the tests and 
the teachers administered them without any meaningful test security. 

In fact, CTB/ McGraw-Hill explicitly instructed the teachers to look at all the 
questions before giving the tests to the students, saying; "arrangements should 
be made so the proctors and examiners actually can take the test." I * 3 Numerous 
teachers - usually my patients - told me they simply memorized or copied the 
test questions and taught their students the answers the following year to ensure 
high scores. If they failed to do so, parents would be angry, colleagues critical, 
newspapers disparaging, and administrators livid. Teacher's annual evaluations 
would inevitably suffer. 

I recently contacted the West Virginia Department of Education to see if anything 

had changed in West Virginia during the last 15 years. I found that from 1996 to 

2002, West Virginia used the exact same test questions for eight years in a row 
[Stanford-9 Achievement Test (SAT-9, form S)]. Scores soared. I dare the reader to 

find even one of the fifty-five West Virginia counties in which the majority of 
elementary students tested below the national average for the four years between 
1999 and 2002. 4 In 2003, West Virginia began excluding fewer lower functioning 
students from test reporting and several dirt-poor counties in the southern 
coalfields fell slightly below the national average. However, in 2003, West 
Virginia statewide SAT-9 scores were still above the national average at all grade 
levels tested. 
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In 2004, West Virginia adopted a criterion-referenced test, the WestTest, in 
consultation with CTB/ McGraw-Hill. I called the Office of Student Assessment 
Services of the West Virginia Department of Education and learned that eighty- 
percent of the WestTest questions are the same year-to-year. 5 I could find 
nothing in their "Testing Code of Ethics" which even suggests that the teachers 
should not look at WestTest question while students are taking the exam. 6 
Using these testing procedures, 78 percent of West Virginia third graders tested 
"at or above mastery" in reading/ language arts in 2005. 7 

However, in 2005, the National Assessment of Educational Progress (NAEP) 
found something quite different. (NAEP is a national test, which samples 
participating school districts in a secure manner.) NAEP reported only 26 
percent - not 78 percent - of West Virginia's fourth graders were proficient in 
reading. 8 It seems that West Virginia students are proficient in West Virginia 
reading but not in American reading! 

Like 20 years ago, the most parsimonious explanation for this discrepancy is that 
teachers in West Virginia are doing today what they were doing twenty years 
ago, memorizing the West Virginia test questions and teaching the answers the 
following year - something they cannot do with the NAEP test. Although I 
didn't know it at the time, and was denied anyway of proving it now, it seems 
likely that West Virginia educators rely on curriculum preparation materials that 
are virtually identical to the WestTest, also something they cannot do with 
NAEP. 

Obviously, this is not occurring in a vacuum. School administrators design 
testing procedures that assure that teachers read the test, but cover their own 
culpability with loosely worded and largely meaningless "Testing Codes of 
Ethics." Although I could not obtain copies of the WestTest to compare those 
items with test or curriculum material, it is my experience - as the reader will see 
- that many schools use test preparation materials that are repeatedly laced with 
the test questions. 

School administrators are the principal beneficiaries of rapidly rising scores and 
they want no part of accurate tests. Administrators need rapidly rising scores to 
show they are doing a good job. In the past, when states like West Virginia 
finally change to an equivalent, but different, test - and teachers see an 
unfamiliar set of questions - scores plummeted, parents complain, newspapers 
criticize, and administrators play musical chairs and exchange jobs. However, 
next year's scores start their inexorable and flattering ascent. 

In the 1980's, school principals in West Virginia often insisted that their teachers 
take the tests beforehand (as the official test manual sometimes suggested) so the 
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teachers had ample time to memorize the questions. When this didn't work, 
some administrators in California- who had physical control of the tests - simply 
erased the student's wrong answers and marked the correct ones. 9 At that point, 
I had no idea of the role that crib-sheet-like test preparation materials played in 
rapidly rising scores. 

By 1985, 1 wanted to know who was supplying the schools with tests that could 
so easily be misused to deceive parents, children, and the community. Posing as 
a school administrator, I called a major test publisher. The woman I spoke with 
was more than happy to supply tests with any "national norms" that I requested 
- all certified by respected testing consultants from major universities. The 
publisher would sell inner city norms, low-socioeconomic norms, adjusted 
norms, etc., with their tests. She explained that choosing the right "national 
norm" was very important. 

If I chose a low performing norm group, I could look forward to high initial 
scores but year-over-year gains would quickly become unbelievably high. If the 
initial norms were "tougher," I would look bad the first year but could look 
forward to very flattering year-over-year gains. The woman finally caught on to 
me when I asked how she knew the scores would go up every year. Of course, 
she knew that unchanging test questions guarantee rising scores. 

By 1986, 1 had found out several important things. The test publishers would 
supply any "national norms" school districts wanted. They also knew that using 
the same test questions year after year assured that enough teachers would read 
the test and cause flattering year-over-year gains. Finally, the publishers were 
making good money selling these "standardized tests," and wanted no part in 
reforming them. What I didn't know is who was consulting for these publishers, 
giving them academic cover? 

In 1987, 1 formed an education reform group. Friends for Education. We 
conducted a series of campaigns to improve schools in West Virginia. For 
example, we held "The Cleanest School in West Virginia Contest." When that 
was ignored, we held a "Dirtiest School in West Virginia Contest," promising a 
bucket, mop and broom to the winner. 10 We held public rallies focusing on 
improving the worst schools in the state. 11 We also filed complaints with the 
Equal Opportunity Employment Commission, claiming school officials were 
illegally denying women administrative positions in West Virginia schools. 12 

However, I kept wondering about the tests. The American educational system is 
built around testing. Could the entire American testing infrastructure be 
corrupt? If West Virginia was testing above the national average, then perhaps 
all the states were reporting the same thing and no one knew it. I called the 
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office of then U.S. Secretary of Education William Bennett only to be told they 
did not collect, nor did they know, scores from individual states. 

Between patients, I had the nurse, x-ray technician, and lab technician from my 
medical clinic call the education departments of all fifty states and collect their 
test scores. For reasons I was beginning to understand, the states and districts 
proved to be quite defensive in releasing scores. Often my clinic staff had to 
make up a compelling reason to get the scores, such as they were thinking of 
moving to the state in question. 

Finally, there it was, I had the data: all fifty states were reporting they were 
above the national average in elementary achievement. I realized for the first 
time that newspapers throughout the country were repeatedly running flattering 
stories on state or local school achievement on one page - and dire warnings that 
the United States was "A Nation at Risk" on another. 

CTB-McGraw Hill wrote me and threatened to sue me should I publish my 
findings. 13 I promptly dipped into my personal savings and published my first 
report: Nationally Normed Elementary Achievement Testing in America's Public 
Schools: How All 50 States Are above the National Average. Although we had no 
money for legal action. Friends for Education immediately sent consumer fraud 
complaints to the Attorneys General in all fifty states, claiming commercial test 
publishers were engaged in deceptive testing practices. 14 My first report showed 
up in headlines around the country, including the front page of the Washington 
Post and an article in the New York Times with the headline: "Standardized Test 
Scores: Voodoo Statistics." 15 ' 16 

The 1988 summer issue Educational Measurement: Issues and Practice, was devoted 
to my first report, with invited responses from the four major commercial test 
publishers, several academics, and the U.S. Department of Education. 17 I 
responded in the winter issue. 18 Two years later, the same journal devoted 
another issue to a U.S. government funded study on my "Fake Woebegone" 
report. It concluded, "The results of the present study provide support for 
Canned's general finding that for the elementary grades almost all states and the 
majority of districts are reporting norm-referenced achievement test results that 
are above the national median." 19 

Eater in 1988, my education about standardized testing in America continued at 
a special meeting of test publishers and academicians at the U.S. Department of 
Education. Secretary William Bennett called the meeting, asking me to explain 
my findings that all fifty states were testing above the national average. It was 
during that meeting Assistant U. S. Secretary of Education Chester Finn first 
labeled my findings, the "Lake Woebegone Report." 
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At the meeting, I learned that a handful of academicians at major American 
universities consulted with test publishers to develop both the tests and the 
various norms. These academics denied that "Lake Woebegone" tests were the 
problem, insisting that educators were misusing their tests due to a "high stakes" 
testing environment. The representatives from the college entrance exam 
publishers, the SAT and ACT, expressed amazement at this explanation. After 
all, their tests suffered from no "Lake Woebegone" psychometrics. 

I did not understand why some of the academics insisted on explaining it was 
due to "high stakes" testing when the obvious problem was a corrupt testing 
infrastructure. If I cheated on my federal tax return, explaining it saved my 
family "high stakes" money, few federal courts would exonerate me under such 
a defense - indeed few defense attorneys would proffer such a meaningless 
defense. 

I left the meeting having learned three things. One, both William Bennett and 
Chester Finn claimed they knew nothing about the cheating. Two, officials from 
the college entrance exams knew about the cheating but were powerless to stop 
it. Three, some of academicians knew all about the cheating and, for reasons I 
didn't understand, wanted "Lake Woebegone" testing to continue. 

By that time, I rightly or wrongly assumed most politicians knew what was 
going on. While speaking at an April 1990 meeting of the Education Writers 
Association in Chicago, I angrily confronted another speaker. Governor Bill 
Clinton. I told Clinton, early in his initial run for the White House, of the 
unbelievably high, rapidly improving, and very politically flattering, test scores 
in illiterate Arkansas. Clinton vehemently denied any knowledge of cheating. 

The following week, Clinton encountered a front-page story in the states' largest 
newspaper about my charges of widespread cheating in Arkansas. 20 Clinton 
then called me and spent thirty minutes asking me questions about things he 
could do to stop the cheating. I told him the keys to preserving the validity of 
the test is changing questions every year, having a large bank of questions, 
maintaining a broad curriculum, testing infrequently, and not focusing on test 
preparation. Another Arkansas newspaper quoted Clinton's response: "When he 
(Cannell) told me that, I said 'Gosh' we'll look into that. It may cost a few more 
thousand dollars but it's worth it if it preserves the integrity of the test." 21 A few 
weeks later, Arkansas announced plans for improvements in test security. 22 In 
1996, then President Clinton went on to recommend a national achievement test 
with strict security - a proposal refused by the Republican Congress. 
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Unlike Clinton, Bush was not spared national media attention about cheating in 
Texas. One month before the election of 2000, the Rand Corporation claimed that 
the "Texas Miracle," the dramatic gains in Texas school achievement that 
propelled George Bush to the White House, were suspect. Rand compared gains 
on the Texas test with Texas gains on National Assessment of Educational 
Progress (NAEP) and concluded that most of the gains on the Texas test were 
bogus. 23 In a separate review of Texas testing, Haney concurred, saying the 
gains "are more illusory than real," and "the Texas Miracle is more hat than 
cattle." 24 


More recently, the Dallas Morning News has uncovered evidence of cheating in 
more than 200 Texas schools and this may be the tip of the iceberg. 25 Although 
the basic statistical technique used to detect cheating is easy to perform, the 
Texas Education Agency chooses not to use it. Furthermore, the Texas Education 
Agency performs erasure analysis (to detect if school officials are erasing the 
kid's wrong answers and filling in the correct ones) but does nothing with the 
information unless they get a complaint! It is interesting to note that Bush, 
unlike Clinton, has failed to propose reforms that would address America's 
corrupt public school testing infrastructure. 

Successful governors, like Clinton and Bush, may have learned long ago of the 
political value of "Lake Woebegone" testing, or, like most Americans, they just 
assume a standardized test is a standardized test. However, public educators, 
test publishers, and academic testing experts cannot claim ignorance - they are 
too intimately involved with the corruption. 

I was hopeful the first "Lake Woebegone" report would reform testing. After all, 
it made headlines around the country. In 1989, when nothing changed, I again 
surveyed all 50 states and published a second report with the help of a grant 
from the Kettering Family Foundation. The "Lake Wobegon" Report, How Public 
Educators Cheat on Standardized Achievement Tests detailed the extent of cheating 
and how to detect it. 

My second report received even more publicity then the first, with front-page 
coverage by the Wall Street Journal. 26 The Rocky Mountain News said I had 
uncovered "the Great Testing Lie." 27 Surely, policy makers would be outraged at 
our corrupt testing infrastructure and realize that American schools would never 
improve until we had honest tests. I then wrote and distributed Testing Ethics 
Model Legislation ; forlornly hoping state legislators would enact simple laws 
reform testing. 28 

No response. However, in 1990 my hopes soared when Sixty Minutes called. 

They were doing a story about teachers and school administrators cheating on 
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tests, highlighting my second report. I was sure that would do it. Scandals 
exposed by Sixty Minutes often led to reform. Morley Safer came to my house 
and brought copies of some "test preparation" materials that CBS News had 
obtained from various states. He asked me to look them over before filming. 
Teacher is a Cheater. 

It hit me like a brick. The "test preparation" materials contained all the answers 
to the test questions; any one who mastered this material would know exactly 
what was on the test. Some of the same academicians who claimed 
"misunderstandings" and "high stakes" testing were the problem, and who 
made money collaborating with test publishers developing "Lake Woebegone" 
tests, had side businesses; they provided school districts with crib sheets. 

I waited for the fallout from the Sixty Minutes report. On March 25, 1990, Sixty 
Minutes ran Teacher is a Cheater and reported the tests were fraudulent and that 
cheating by educators was rampant in American schools. 29 The cheating took 
many forms, from outright falsification of children's answer sheets and teachers 
memorizing test questions, to the most insidious - closely aligning a narrow 
curriculum with a dumbed-down test while using crib-sheet-like test preparation 
materials. Sixty Minutes, like the Wall Street Journal, reported the most common 
corruption was assuring that scores steadily climbed due to crib-sheet-like test 
preparation-materials. 30 

However, nothing happened. The Sixty Minutes report provoked no outrage, no 
commissions, no hearings, nothing. I could do no more. I resigned from Friends 
for Education and the organization fell apart. I quickly became involved with 
two entirely different causes: first, false recovered memories of sexual abuse, and 
subsequently, widespread vitamin D deficiency. 31 

My education reform days are painful to remember, mainly because I fear that 
little has changed and my work was for naught. Every year, from my California 
home, I read hundreds of stories online, published in local California 
newspapers, all about how test scores are improving in their local schools. The 
same articles praise California politicians and the California State Department of 
Education for statewide improvements. 

As part of their state-run STAR testing program, the California Department of 
Education administered the Stanford-9 from 1998 to 2002, using the same 
booklets - and the same questions - for five years in a row. 32 California fourth 
grade national percentile rank reading scores on the Stanford-9 section of the 
California STAR testing program are instructive: 


Year 


Reading Language 
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1998 

40 

4433 

1999 

42 

46 34 

2000 

45 

50 35 

2001 

47 

53 3 6 

2002 

50 

55 37 


Pretty impressive. In five years, California went from among the lowest reading 
and language scores in the country to at or above the national average. 

However, in 2003, California changed to the California Achievement Test-6 
(CAT-6), and scores plummeted when the teachers encountered unfamiliar test 
questions. Fourth grade California reading scores fell to 39 and language scores 
fell to 42, lower than they were in 1998. 38 


In 2002, California began to emphasize the other component of their STAR 
testing program; the state developed, California Standards Test (CST). I 
understand, but the California Department of Education will neither confirm nor 
deny, that about 50% of the questions on the CST are the same year-to-year. 
Fourth grade mean scaled English/ Language Arts scores on the CST give a 
similar, but less dramatic, impression of improving learning in California. 39 

2002 333 

2003 339 

2004 339 

2005 346 

Compare the Stanford-9 and CST to the fourth grade NAEP reading scores for 
California. Examine the percentage of California children who performed at or 
above the rudimentary "basic" reading level on NAEP: 40 
1998 48 

2002 50 

2003 50 
2005 50 

NAEP reports no significant difference in fourth grade reading average scale 
scores during the same time although increasing participation rates may have 
masked any gain. A Even more telling is California's NAEP ranking: NAEP 


A Comparing NAEP scores over time is problematic because of varying participation rates. California certainly performs 
poorly on NAEP, but NAEP refuses to correct itself for changes in exclusion and accommodation rates. In 1998, 

California excluded 14 percent of fourth grade reading students as English Language Learners (ELL) or students with 
learning disabilities (SD). In 2005, the number dropped to 5 percent. With more weak students now in its NAEP sample, 
California's scores might have dropped and they didn't. One has to consider the NAEP exclusion and accommodation 
rates before comparing year over year changes in NAEP. 
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reports that of the 52 other states and jurisdictions that participated in the 2005 
assessment, California performed better than only one other jurisdiction. 

The Thomas B. Fordham Foundation recently came to the same conclusion. They 
listed California as one of several states with rather dramatic gains on state 
controlled tests, but no progress on NAEP testing . 41 The same study found 
Tennessee was the most brazen. In 2005 , eighty-eight percent of Tennessee 
eighth-graders tested proficient on their "Lake Woebegone" state reading test, 
while only 26 % were proficient on NAEP. That is, virtually all Tennessee 
students are proficient in Tennessee reading, but very few in American reading. 

Fifteen years after "Lake Woebegone" testing was uncovered, California, the 
largest state in the union, continues to conduct corrupt testing. Although I was 
not able to learn if academics are still supplying "test preparation materials" to 
California schools, it matters little. California teachers tell me they really don't 
have to memorize the test questions anymore, the curriculum materials supplied 
by the state are laced, repeatedly, with the test questions. 

Recent federal legislation has undoubtedly made the cheating worse because No 
Child Left Behind (NCLB) increased the consequences of the tests but Bush and the 
Congress failed to reform America's testing infrastructure. Furthermore, my 
experience with recent graduates from American schools of education is that 
they are misinformed about the essence of testing (accurate sampling) and 
pathologically opposed to virtually all forms of testing. Such mindsets are fertile 
fields for widespread corruption. 

However, it is unfair to blame NCLB for corrupt testing; it existed long before 
NCLB. Two recent press articles detailed widespread cheating on NCLB testing, 
blaming it on the "high stakes" testing environment created by NCLB . 42 ' 43 
However, the reporters failed to note that the media reported on widespread 
cheating in state run testing programs well before NCLB. 

44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60 what NCLB did was greatly increase the stakes 
of testing, while relying on a corrupt testing infrastructure to measure results. 

When NAEP scores are lower than NCLB state scores, state superintendents of 
schools never mention their crib-sheet-like test preparation material, their lack of 
outside proctors, or the fact that teachers know what is on the test because many 
of their questions are the same every year. Instead, they simply explain that they 
have not aligned their curriculum with the NAEP test. 

California Superintendent of Schools, Jack O'Connell, recently explained to the 
Los Angeles Times, "Results on our statewide tests, which are aligned to our 
rigorous standards, indicate that a focus on high expectations is leading to steady 
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gains in student achievement." 61 He failed to mention that 50% of the questions 
on the statewide test are the same year-to year, that test preparation materials are 
laced with test questions, and California fails to use independent test proctors. 
Although less dramatic then West Virginia and Tennessee, fourth grade students 
in California show steady improvement in California reading but not in 
American reading. 

As Samuel Johnson said in Boswell's Life, the solution "is as well-known, and has 
long been as well known, as ever it will be." The vast majority of test questions 
must be different every year, questions must come from a large bank of 
questions, test preparation materials should be minimal, and outside proctors, 
not school officials, should administer the tests. The Director of the Oklahoma 
State Bureau of Investigation, Robert Hicks, said it well. 

In responding to Friends for Educations' charges of widespread cheating in 
Oklahoma, Hicks wrote to then Governor Henry Bellman, who wanted to know 
if the state should prosecute the cheating teachers. Director Hicks advised 
against it. He explained, "The tests lend themselves to being compromised by 
using the same questions repeatedly for several years and also allowing the tests 
to be administered by school officials." 62 

I agree. It is unfair to blame only the educators for a corrupt testing 
infrastructure, an infrastructure now required by the federal government. If I 
were a state superintendent of schools in one of the 25 lower-performing states, I 
would demand a test that showed we were meeting NCLB standards and the 
easiest way to do that is to use the same questions year after year or to produce 
test preparation materials or curriculum materials that are laced with test 
questions. Likewise, if I was a county or district superintendent of schools, I 'd 
make sure my principals had easy access to the tests and concentrated on test- 
question-laced-curriculum materials. If I were a principal. I'd do the same. If I 
were a teacher. I'd either carefully study the exam questions and, the next year, 
teach the children enough of the answers to be sure I could meet NCLB 
standards or narrow my curriculum to the corrupt test preparation/ curriculum 
materials. I cannot blame educators for doing what I would do, where I caught 
in a corrupt system. 

In their lives, my two young daughters will compete with children from other 
countries who have been ruthlessly educated. My daughters need a broad-based 
education driven by an incorruptible test. As someone once said, "If the 
educators want to know what will be on the test, tell them the English section 
will have lots of letters and the math section lots of numbers." 
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Instead of preparing children with a broad and challenging curriculum, 
American schools teach a narrow curriculum driven by corrupted tests and we 
are now doing so in the name of leaving no child behind. Like the "Lake 
Woebegone" testing that preceded it, NCLB testing mollifies parents, 
compliments educators, promotes political careers, enriches publishers, and 
ensures profitable consultation fees to academics at many of America's major 
schools of education. Until we repair America's corrupt testing infrastructure, 
American schools will continue to flounder in a sea of mediocrity. 
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